One major objective of computer vision algorithms is to create spatial and temporal consistent 3D information from multiple cameras and/or multi-sensor setups.
Patch sweeping is an extension of the well-known plane sweeping approaches. In recent years, plane sweeping was considered as one of the most promising approaches for real-time depth estimation, but the estimation quality is limited (Colins, R. T. (1996): “A Space-Sweep approach to true Multi-Image matching”, Computer Vision and Pattern Recognition, IEEE Computer Society Conference, vol. 0 p. 358).
Simultaneously, patch based techniques (Furukawa, Y. Ponce, J. (2007): “Accurate, dense, and robust Multi-View stereopsis”, Computer Vision and Pattern Recognition, CVPR 07) turned out to be among the most accurate but time consuming 3D reconstruction methods (Seitz, S. M., Curless, B., Diebel, J., Scharstein, D., Szeliski, R. (2006): “A comparison and evaluation of Multi-View Stereo reconstruction algorithms”, Proceedings of the 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, IEEE Computer Society). Inspired by the contrary properties of the plane sweeping and patch based approaches, patch sweeping emerged as a symbiotic fusion of these two techniques.
Patch sweeping assumes that a 3D object surface can be described with quadratic surface elements, which are also known as surflets or spatial patches. In order to estimate the depth of an object for a reference camera or the object surface, several oriented spatial patches along the viewing rays of this reference camera have to be tested (see FIG. 1).
FIG. 1 schematically illustrates the principle of Patch Sweeping. Several surflet orientations are evaluated.
An extended hybrid recursive matching (HRM) was proposed by N. Atzpadin, P. Kauff, O. Schreer in 2004 in the article “Stereo Analysis by Hybrid Recursive Matching for Real-Time Immersive Video Conferencing”, IEEE Trans. on Circuits and Systems for Video Technology, Special Issue on Immersive Telecommunications, Vol. 14, No. 4, 321-334. European Patent Application EP 1 229 741 A1 by Atzpadin, N., Karl, M., Kauff, P., Schreer, O. (2002) describes a “Process for Real-Time Correspondence Analysis of Image Features in Corresponding Video Images”. The article “True-Motion Estimation with 3-D Recursive Search Block Matching” by De Haan, G., Biezen, P. W. A. C., Huijgen, H., Ojo, O. A., IEEE Trans. on Circuits and Systems for Video Technology, (1993), Vol. 3, No. 5, 368-379 describes further related research.
One major challenge of conventional patch sweeping is to define the search range in the parameter space in order to get results of high depth resolution but with a reasonable computational effort.